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ABSTRACT 


Computational  experimentation  is  an  important  tool  of  the  military.  It 
provides  useful  insights  at  a  lower  cost  of  time  and  money,  when  compared  to 
physical  experiments.  Consequently,  computational  experiments  are  used  to 
evaluate  weapon  systems  for  technology  acquisition,  examine  tactics,  and  to 
help  select  among  alternatives  for  military  operations  and  war  plans. 

Experiments  often  consist  of  a  large  number  of  factors.  Advancements  in 
computing  power  and  design  of  experiments  (DOE)  for  simulation  allow  for  the 
investigation  of  more  of  these  factors  through  computational  experiments, 
achieved  with  less  expense  in  time,  effort,  and  money.  Within  the  framework  of 
DOE,  this  thesis  investigates  Orthogonal  Latin  Hypercube  (OLH)  and  Nearly 
Orthogonal  Latin  Hypercube  (NOLH)  designs.  These  designs  are  often  used  for 
computational  experiments.  This  research  greatly  expands  upon  the  size  (in 
terms  of  runs  and,  especially,  variables)  of  the  available  OLH  and  NOLH  designs. 
Previously,  the  largest  catalogued  OLH  and  NOLH  designs  were  a  maximum  of 
29  variables  and  257  runs.  OLH  and  good  space-filling  NOLH  designs  for  up  to 
512  variables  in  1025  runs  are  now  available.  This  thesis  also  develops  an 
algorithm  for  handling  discrete  factors  with  the  designs.  Finally,  the  effects  of 
stacking  multiple  OLH  designs  into  one  larger  design  are  quantified.  All  of  the 
designs  developed  in  this  research  are  available  at  the  Simulation  Experiments  & 
Efficient  Designs  (SEED)  Center  website  (http://harvest.nps.edu). 
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EXECUTIVE  SUMMARY 


The  Simulation  Experiments  &  Efficient  Designs  (SEED)  Center  of  the 
Naval  Postgraduate  School  (NPS)  has  a  mission  to  “promote  the  advancement  of 
collaborative  development  and  the  use  of  simulation  and  efficient  designs  to 
provide  decision  makers  with  timely  insights  on  complex  systems  and 
operations.” 

One  of  the  most  widely  used  designs  for  computational  experiments  at  the 
SEED  Center  and  elsewhere  is  the  Latin  Hypercube  (LH)  design.  LH  designs 
are  attractive  for  computational  experiments  because  they  allow  analysts  to 
efficiently  vary  many  factors  simultaneously.  Furthermore,  LH  designs  provide  a 
lot  of  flexibility  for  subsequent  analysis. 

For  any  given  combination  of  input  factors  and  runs,  there  exist  a  large 
number  of  possible  LH  designs.  From  among  the  possibilities,  LH  designs  that 
are  orthogonal  and  with  good  space-filling  properties  are  preferred.  Designs  with 
good  orthogonal  properties  are  desirable  because  they  provide  uncorrelated 
estimates  for  the  regression  coefficients  that  analysts  frequently  use  to  analyze 
simulation  models.  Good  space-filling  properties  are  useful  for  data  analysis 
across  the  possible  range  of  data  because  the  experimental  points  are  scattered 
throughout  the  domain  in  a  uniform-like  manner,  providing  information  throughout 
the  design  space.  LH  designs  which  are  orthogonal  can  have  poor  space-filling 
properties.  Thus,  a  special  class  of  LH  designs,  called  Nearly  Orthogonal  Latin 
Hypercube  (NOLH),  which  are  nearly  orthogonal  but  with  improved  space-filling 
properties,  were  developed  by  Cioppa  (2002). 

Existing  catalogued  Orthogonal  Latin  Hypercube  (OLH)  and  NOLH 
designs  can  handle  up  to  29  factors  in  257  runs.  In  this  thesis,  the  number  and 
size  of  designs  currently  used  by  the  SEED  Center  have  been  dramatically 
increased.  Specifically,  the  readily  available  OLH  and  NOLH  designs  have  been 


XV 


increased  to  512  input  variables  and  1025  design  points.  Table  A  shows  how 
this  increases  from  the  previous  works  of  Ye  (1998)  and  Cioppa  (2002). 


m 

N 

k  (Ye) 

k  (Cioppa) 

k(Ang) 

4 

17 

6 

7 

8 

5 

33 

8 

11 

16 

6 

65 

10 

16 

32 

7 

129 

12 

22 

64 

8 

257 

14 

29 

128 

9 

513 

16 

37 

256 

10 

1025 

18 

46 

512 

Table  A.  Maximum  number  of  input  variables  for  OLH  and  NOLH 
designs  from  Ye  (1998),  Cioppa  (2002),  and  Ang  (2006). 

By  comparing  the  properties  of  LH  designs  with  different  construction, 
some  of  the  findings  include: 

Florian’s  method,  used  in  the  construction  of  NOLH  designs, 
improves  the  orthogonal  properties  of  the  LH  designs. 

Increasing  the  combinations  of  permutation  matrices,  to  increase 
the  number  of  factors  with  the  same  number  of  runs,  tends  to  cause 
downgraded  orthogonal  and  space-filling  properties. 

LH  designs  assume  that  the  input  variables  are  continuous.  The  use  of 
discrete  variables  affects  the  orthogonal  and  space-filling  properties  of  the 
designs.  This  thesis  provides  a  useful  guide  for  choosing  columns  for 
incorporating  discrete  variables  into  the  catalogued  OLH  and  NOLH  designs. 
This  guide  provides  designs  with  the  best  orthogonal  and  space-filling  properties. 
Some  of  the  findings  include: 

Positive-Negative  coding  is  best  suited  for  2-level  variables. 

The  orthogonal  properties  are  affected  for  discrete  variables  with 
fewer  levels,  when  compared  to  discrete  variable  with  more  levels. 
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One  efficient  way  to  improve  the  space-filling  of  an  LH  design  is  to  stack 
several  designs  together.  While  this  has  been  done  in  the  past,  the  effects  have 
not  been  quantified.  This  thesis  quantifies  the  improvement  of  the  orthogonal 
and  space-filling  properties,  in  relation  to  the  number  of  stacks,  for  the  various  LH 
designs.  The  LH  generator  developed  in  the  thesis  allows  for  the  flexible  creation 
of  whatever  stacked  designs  a  user  desires. 

Users  of  the  LH  designs  can  benefit  from  the  larger  designs  developed, 
and  a  clearer  understanding  of  the  effects  of  discrete  variables  and  stacking  on 
the  orthogonal  and  space-filling  properties  of  the  LH  designs.  These  designs 
and  the  JAVA  generator  needed  to  construct  new  OLH  and  NOLH  designs  are 
available  at  the  SEED  Center  website  (http://harvest.nps.edu). 
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I.  INTRODUCTION 


A.  FRAMING  THE  PROBLEM 

Computer  simulation  and  experimentation  is  important  to  the  military.  It  is 
used  in  the  evaluation  of  weapon  systems  and  technology  prior  to  acquisition, 
providing  the  ability  to  analyze  boundary  and  dangerous  conditions  that  are 
difficult  to  replicate  or  are  hazardous  in  physical  experiments.  Computer 
simulation  and  experimentation  of  military  operations  and  war  plans,  with  military 
simulation  models  like  JCATS''  and  MANA2,  allow  military  decision  makers  to 
study  the  effects  of  tactics,  employment  options,  mix  of  forces,  weapon 
acquisition,  and  testing  of  war  plans.  Through  computer  simulation  and 
experimentation,  there  are  also  savings  in  material,  time  and  money. 

Military  uses  of  computer  simulation  and  experimentation  range  from  the 
acquisition  of  the  5.56  mm  round  to  the  testing  of  war  plans,  and  each  simulation 
or  experiment  contains  numerous  factors.  These  inputs  tend  to  be  uncertain 
(i.e.,  the  performance  of  humans  and  systems  in  battle)  and  it  is  unreliable  to  use 
assumptions  of  strong  prior  knowledge  (Cioppa  &  Lucas,  2006).  Unfortunately, 
the  total  enumeration  of  all  possible  scenarios  for  experimentation  is  an 
exhaustive  task  even  with  today’s  computing  power. 

In  the  field  of  design  of  experiments  (DOE),  there  are  effective  and 
efficient  methods  to  design  computational  experiments.  Specific  DOE  for 
computer  simulation  and  experimentation,  an  extension  from  the  traditional  DOE, 
has  been  proposed  (Kleijnen,  Sanchez,  Lucas,  &  Cioppa,  2005). 

Nearly  Orthogonal  Latin  Hypercube  (NOLH)  designs  possess 
improvements  in  space-filling  properties  with  nearly  orthogonal  properties  for 
Latin  Hypercube  (LH)  designs  (Cioppa,  2006),  (Cioppa  &  Lucas,  2006).  NOLH 
designs  have  been  commonly  used  for  military  simulation  experiments  since  their 


1  Joint  Conflict  and  Tactical  Simulator  (JCATS)  http://www.jfcom.mil/about/factJcats.htm 
(accessed  27  May  2006). 

2  Map  Aware  Non-uniform  Automata  (MANA),  developed  for  the  New  Zealand  Army  and 
Defense  Force. 
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inception  (See  SEED  Center  website  at  http://harvest.nps.edu  for  more  details). 
Current  designs  readily  available  include  NOLH  designs  that  can  handle  up  to  29 
factors  with  257  runs. 

This  thesis  extends  the  catalogue  of  the  designs  up  to  512  factors  with 
1025  runs.  The  effects  of  using  discrete  variables  with  LH  designs  are  also 
investigated,  and  methods  for  obtaining  LH  designs  with  good  orthogonal  and 
space-filling  properties  are  proposed.  The  effects  of  stacking  the  LH  designs  on 
the  orthogonal  and  space-filling  properties  of  the  designs  are  also  investigated 
and  quantified. 

B.  EXPERIMENTS 

By  conducting  experiments  through  computer  simulation  and 
experimentation,  one  can  explore  a  much  greater  number  of  factors  than  is 
feasible  through  traditional  physical  experimentation.  Not  only  does  the  use  of 
computer  experimentation  provide  more  consistent  replications,  but  it  is  a  great 
deal  more  cost  efficient  than  traditional  methods  of  physical  experimentation. 

Computer  simulation  and  experimentation  are  used  to  compare  the 
performance  of  systems.  With  traditional  experiments,  there  are  numerous 
methods  developed  to  detect  input  variables  that  may  or  may  not  cause  some  of 
the  variation  in  the  responses.  This  includes  randomization,  blocking,  and 
replication.  In  contrast,  the  use  of  computer  experiments  removes  the 
requirement  for  these  three  methods,  as  computer  experiments  provide  identical 
answers  for  repeated  runs  with  the  same  set  of  inputs,  reducing  the  need  for 
randomization,  blocking,  and  replication  (Banter,  Williams,  &  Notz  ,  2003). 

An  experiment  consists  of  responses  due  to  two  main  types  of  variables, 
controlled  variables  and  environmental  variables.  Controlled  variables  are 
variables  that  can  be  set  to  “control”  the  product  or  process.  Environmental 
variables  are  the  “uncontrollable”  conditions  under  which  the  product  or  process 
is  executed.  An  example  of  a  military  experiment  is  the  determination  of  the 
accuracy  of  a  tank  gun:  the  controlled  variables  include  the  weapon  system. 
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ammunition  type,  target  types  and  the  operating  crew;  the  uncontrolled  variables 
include  the  weather  conditions.  The  response  of  the  experiment  is  the  distance 
of  the  impact  point  from  the  aim  point. 

A  metamodel  is  often  used  to  estimate  the  effects  of  the  input  variables  on 
the  responses.  The  utility  of  metamodels  is  that  they  allow  the  investigation  to 
focus  on  the  important  factors  that  may  influence  the  experiment.  They  are  then 
used  as  quick  turn-around  estimates  of  the  actual  problem  and  to  replace  running 
the  actual  simulation,  which  generally  takes  a  long  time  to  generate  results.  A 
description  of  metamodels  can  be  found  in  subsection  1 . 

The  DOE  Toolkit  (Kleijnen,  Sanchez,  Lucas,  &  Cioppa,  2005),  which 
provides  designs  that  are  used  to  investigate  computer  simulation  and 
experiments,  can  be  found  in  summary  in  subsection  2. 

1.  Metamodel 

A  metamodel  describes  the  relationship  between  the  output  response 
(often  denoted  as  y)  and  the  k  input  variables  (often  denoted  as  A 

metamodel  is  a  function  g  that  can  be  estimated  given  an  experimental  design 
and  the  corresponding  responses,  as  described  in  Equation  1.1. 

y^=g{x„X2,...,x,)  (1.1) 

The  multiple  regression  modeP  is  often  used  as  the  metamodel  for 
exploration,  where  the  inputs  for  the  regression  coefficients  are 

usually  estimated  from  computer  simulation  and  experiments  by  using  the 
method  of  least  squares.  The  error  term,  s ,  is  often  assumed  to  be  independent 
and  identically  distributed  (/./.d.)  with  expectation,  E{s)  =  0,  and  variance, 

Var{£)  =  (7^ . 

The  multiple  regression  model  with  k  predictor  variables  is 

J<  ^  i.i.d 

g(x^,..,x^)  =  +^y0,.x,.  +£  Where  e  ~  Normal{0,a^)  (1.2) 

/=i 

3  Montgomery,  et  al.  (2001),  Chapter  3. 
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Metamodels  with  more  complex  structures,  such  as  the  cubic  polynomial 
model,  y  =  +  +  p^x^  +  a  can  be  represented  by  Equation  1 .2.  This  is 

achieved  by  the  assignments  x^  =x,  x^=  x^,  and  Xg  =  .  Similarly,  models  with 

interactions,  i.e.,  y  =  Pq+  P^x^  +  p^x^  +  p^^x^x^+s ,  and  quadratic  polynomial  with 
interactions  models,  i.e.,  y  =  p^+  p^x^  +  Pyx^  +  p^^x\  +  yS|2X.,X2  +  s ,  can  also  be 
represented  by  Equation  1.2. 

2.  Design  of  Experiments 

The  field  of  DOE  has  a  rich  history  in  a  range  of  fields,  from  farming  to 
medicine,  and  provides  for  the  efficient  conduct  of  experiments.  Computer 
simulation  and  experimentation  provide  a  vast  area  in  which  to  expand  DOE. 
Some  of  the  possible  designs  are  proposed  by  the  DOE  Toolkit  (Kleijnen, 
Sanchez,  Lucas,  &  Cioppa,  2005)  shown  in  Figure  1: 


Many 


o 

o 

« 

o 

I 


Few 

Figure  1 .  Recommended  designs  from  DOE  Toolkit  (Kleijnen,  Sanchez,  Lucas, 

&  Cioppa,  2005). 
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The  number  of  factors,  along  the  vertical  axis,  reflects  the  number  of 
possible  “controlled”  factors  in  the  simulation  experiment.  Typical  simulation 
experiments  consist  of  large  numbers  of  potential  input  factors.  The  toolkit 
describes  different  designs  for  different  applications. 

The  response-surface  complexity,  along  the  horizontal  axis,  defines  the 
continuum  from  simple  to  complex  response  surfaces,  which  are  affected  by  the 
assumptions  on  the  complexity  of  the  metamodel.  “Maximal  screening”  is  used 
when  the  assumed  metamodel  is  simple,  i.e.,  with  no  interactions  or  polynomial 
terms,  and  iJ.d.  errors.  “Minimal  assumptions”  allow  for  more  complex 
metamodels.  The  horizontal  axis  lists  these  assumptions  as  guidance  for  the 
choice  of  the  designs  that  can  be  used. 

LH  designs  are  recommended  for  simulation  experiments  with  large 
numbers  of  factors  and  minimal  assumptions.  For  the  assumption  of  a  linear 
model.  Ye  developed  an  algorithm  (Ye,  1998)  for  orthogonal  LH  designs  which 
were  subsequently  extended  by  Cioppa,  who  exchanged  small  amounts  of  non¬ 
orthogonality  for  better  space-filling  and  developed  the  NOLH  designs  (Cioppa, 
2002).  This  thesis  furthers  the  current  research  on  these  OLH  and  NOLH 
designs. 

C.  RESEARCH  GOALS 

Construction  of  the  OLH  and  NOLH  designs  requires  a  large  amount  of 
computation,  computer  memory,  and  time.  There  are  currently  a  limited  number 
of  catalogued  designs,  and  this  thesis  greatly  expands  the  library  of  the  OLH  and 
NOLH  designs  to  accommodate  greater  numbers  of  variables  and  runs. 

Traditional  LH  designs  have  been  used  with  the  assumption  of  continuous 
variables.  However,  real  world  variables  often  do  not  take  on  continuous  values, 
rather,  they  take  on  discrete  values.  The  orthogonal  and  space-filling  properties 
of  the  OLH  and  NOLH  designs  are  diminished  when  used  with  discrete  variables. 
The  thesis  explores  the  effects  of  this,  and  provides  a  general  rule  of  thumb 
when  using  the  LH  designs  for  different  numbers  and  levels  of  discrete  variables. 
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D.  SOFTWARE 

JAVA  is  used  as  the  programming  language  to  generate  the  LH  designs  in 
this  thesis.  The  code  was  initially  developed  to  be  run  within  the  netBeans  IDE 
5.04  running  J2SE  5.0.  During  the  course  of  the  development,  Eclipse  SDK  3.2 
for  Windows  was  subsequently  used.  In  addition,  an  open  source  COLT 
package  was  used  to  supplement  the  existing  JAVA  packages.  COLT  provides  a 
set  of  Open  Source  Libraries  for  High  Performance  Scientific  and  Technical 
Computing  in  Java.^ 

With  the  COLT  package,  the  License  Agreement  is: 

Copyright  (c)  1999  CERN  -  European  Organization  for  Nuclear 
Research. 

Permission  to  use,  copy,  modify,  distribute  and  sell  this  software 
and  its  documentation  for  any  purpose  is  hereby  granted  without 
fee,  provided  that  the  above  copyright  notice  appear  in  all  copies 
and  that  both  that  copyright  notice  and  this  permission  notice 
appear  in  supporting  documentation.  CERN  makes  no 
representations  about  the  suitability  of  this  software  for  any 
purpose.  It  is  provided  "as  is"  without  expressed  or  implied 
warranty. 


E.  THESIS  ORGANIZATION 

This  thesis  extends  previous  work  on  OLH  and  NOLH  designs.  Chapter  II 
focuses  on  the  DOE  toolkit,  in  particular,  the  orthogonal  and  space-filling 
properties  of  LH  designs,  and  the  construction  of  OLH  and  NOLH  designs. 
Chapter  III  investigates  the  properties  of  the  LH  designs  with  respect  to 
categorical  and  mixed-level  variables.  Chapter  IV  covers  the  results  from  the 
research.  Chapter  V  concludes  and  recommends  future  research. 


4  netBeans  5.0  and  J2SE5.0  Update  6  is  available  from  Sun 
http://java.sun.eom/j2se/1.5.0/download.jsp  (last  accessed  29  May  2006). 

^  COLT  package  from  http://dsd.lbl.gov/~hoschek/colt/  (last  accessed  29  May  2006). 
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II.  DESIGN  OF  EXPERIMENTS  FOR  COMPUTER  SIMULATION 

AND  EXPERIMENTATION 


A.  LATIN  HYPERCUBE  DESIGN  MATRIX 

A  design  matrix,  X,  specifies  how  an  experiment  is  to  be  conducted,  as 
illustrated  in  Figure  2.  The  rows  of  the  matrix,  n,  represent  the  different  design 
points,  or  runs,  for  the  experiment.  Each  row  represents  a  single  run  of  the 
experiment.  The  columns,  k,  represent  the  different  factors  or  controlled 
variables  for  the  experiment.  The  nxk  design  matrix  is  used  as  inputs  for  the 
experiment,  and  the  responses  of  the  experiment  are  recorded  as  the  experiment 
occurs. 


The  values  in  the  matrix,  represented  by  {row,  column),  state  the  value  for 
a  specific  variable  during  a  specific  run  of  the  experiment.  For  example,  the 
representation  {1,  x1)  reflects  that  the  value  of  variable  x1  for  run  1  is  set  at  one. 
The  notation  we  use  to  represent  a  design  matrix  is  . 


n 


< - k  - ► 
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run 
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x2 

x3 

1 

1 
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1 
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1 

0 
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1 
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1 

0 

0 

5 

0 

1 

1 

6 

0 

1 

0 

7 

0 

0 

1 

8 

0 

0 

0 

(a) 


run 

x1 

x2 
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2 
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4 

4 

3 

(b) 


Figure  2.  Design  matrices:  (a)  A  3-factor,  2-level  Full-Factorial  Design;  (b)  A  2- 

factor  Latin  Hypercube  Design. 


7 


A  Latin  Hypercube  (LH)  design  is  represented  in  Figure  2(b).  The 
structure  of  LH  designs  require  the  columns  to  be  permutations  of  the  integers 
from{1,2,...,n} ,  as  shown  byX^  in  Figure  2(b). 

B.  LATIN  HYPERCUBE  SAMPLING 

Latin  Hypercube  Sampling  (LHS)  is  a  statistical  sampling  method  to 
generate  inputs  for  computer  experiments  (McKay,  et  al.,  1979).  The  inputs  for 
the  computer  experiments  can  be  represented  as  either  a  Latin  Square  for  two 
dimensions  or  an  LH  design  for  an  arbitrary  number  of  dimensions. 

Orthogonal  Latin  Hypercube  (OLH)  and  Nearly  Orthogonal  Latin 
Hypercube  (NOLH)  designs  are  special  classes  of  LH  designs.  OLH  designs 
have  strict  orthogonal  properties,  reflected  by  the  condition  number  and 
maximum  pairwise  correlation  of  the  design.  Both  measures  are  described  in 
Section  C.  An  OLH  design  has  a  condition  number  of  1  and  a  maximum  pairwise 
correlation  of  0. 

The  algorithm  by  Ye  (1998)  for  construction  of  OLH  designs  can  be  found 
in  Section  D. 

NOLH  designs,  a  term  coined  by  Cioppa  (2002),  relaxes  the  requirements 
on  the  orthogonal  properties.  He  chooses  the  most  space-filling  design  amongst 
a  few  design  matrices  that  satisfy  the  near  orthogonal  thresholds.  Typical  limits 
used  by  Cioppa  are  a  maximum  pairwise  correlation  of  no  greater  than  0.03  and 
a  conditional  number  no  greater  than  1.13. 

Cioppa’s  8-step  algorithm  to  construct  NOLH  designs  can  be  found  in 
Section  E. 

C.  DESIRABLE  CHARACTERISTICS 

1.  Orthogonal  Measures 

Designs  with  good  orthogonal  properties  are  desirable  because  they 
provide  uncorrelated  estimates  for  the  regression  coefficients  that  analysts  use  to 
analyze  simulation  models.  Two  of  the  orthogonal  measures  used  to  examine  a 
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design  matrix  include  the  absolute  maximum  pairwise  correlation,  ,  and  the 
condition  number,  cond{X^ X) . 

a.  Absolute  Maximum  Pairwise  Correlation, 

The  absolute  maximum  pairwise  correlation  (Hernandez,  2006)  is 
the  maximum  |/?^|,  over  all  Z,),  where  /  ^ and  is  the  pairwise  correlation 

between  columns  x,  and  xj.  The  equation  for  correlation  between  any  two  column 
vectors  ( X'  and  X‘ ),  where  X,'  and  Xj  denotes  the  entry,  while  X'  and  X‘  are 
the  means  of  the  values  in  the  columns,  is  given  in  Equation  2.1. 

-X‘) 

P„  =  I  (2.1 ) 

V  1=1  /=! 

Pamp  gives  the  worst  correlation  amongst  all  pairs  of  columns  in  the 
design  matrix.  A  p^^^  of  0  reflects  an  orthogonal  design  matrix.  As  a  measure  of 
near  orthogonality  for  a  design,  a  smaller  value  is  preferred. 

b.  Condition  Number,  cond{X^X) 

For  an  X"  design,  cond{X^ X)=\\fd\\fn,  where  \\^^  and  \|/n  are  the 

largest  and  smallest  eigenvalues  of  X^X,  when  the  columns  are  centered  to 
sum  to  zero  and  scaled  to  the  range  [-1,1]. 

An  orthogonal  design  matrix  has  a  condition  number,  cond{X^ X), 
equal  to  1.  As  a  measure  of  near  orthogonality  for  a  design,  a  condition  number 
as  close  to  one  as  possible  is  preferred. 

2.  Space-Filling  Measures 

Designs  with  good  space-filling  properties  are  useful  for  data  analysis 
across  the  possible  range  of  data  because  the  experimental  points  are  scattered 
throughout  the  domain  in  a  uniform-like  manner.  Three  measures  to  assess  the 
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space-filling  properties  of  a  design  matrix  are  the  modified  L2 -discrepancy  {ML2), 
the  centered  /.j -discrepancy  (C/.2),  and  the  Euclidean  maxi-min  {Mm)  distance. 

a.  Modified  L2-Discrepancy,  I\/IL2 

The  /.p -discrepancy  is  a  measure  of  the  non-uniformity  of  a  design 

(Wei  Chen  &  Sudjianto,  2003).  The  ML2  is  a  variant  of  the  /.p -discrepancy  and 

has  been  used  as  a  space-filling  measure  in  the  construction  of  NOLH  designs 
(Cioppa,  2002).  The  measure  used  by  Cioppa  is  Equation  2.2,  and  the  same 
measure  in  the  original  source  (Hickernell,  1998)  is  Equation  2.3. 


MmX)  = 


v3. 


2^-k 


n 


111(3 -x^,) 


cf=1  i=^ 


-max(x^,,x^j)] 


d=1  j=1  (=1 


(2.2) 


[ML,{X)f 


v3. 


2 

n 


cf=1  /=1 


-max(x^,,x^y)] 


d=^  i=1  i=1 


(2.3) 


The  two  equations  differ  in  two  ways:  the  equation  2.3  reflects  that 

2 

ML2  is  a  square  root  of  Right  Hand  Side  (R.H.S.)  of  the  equation,  and  has  —  in 

n 

place  of - .  Equation  2.2  is  used  in  this  thesis  with  the  values  normalized  to 

n 

[0,1]  in  each  dimension,  to  allow  for  consistency  with  the  efforts  on  NOLH 
designs  (Cioppa,  2002).  As  a  space-filling  measure  for  a  design,  a  smaller  value 
for  ML2  is  preferred. 

b.  Centered  L2-Discrepancy,  CL2 

The  CL2  is  another  variant  for  the  /.p -discrepancy  that  measures 

space-filling.  The  main  difference  from  ML2,  which  is  anchored  at  the  origin,  is 
that  C/.2  anchors  at  the  center  of  the  hypercube.  The  equation  2.4  (Fang,  et  al., 
2000)  is  used  with  the  values  normalized  to  [0,1]  in  each  dimension.  As  a  space¬ 
filling  measure  for  a  design,  a  smaller  value  for  CL2  is  preferred. 
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(2.4) 
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c.  Euclidean  Maxi-min  Distance,  Mm 

Mm  is  the  other  space-filling  measure  used  by  Cioppa  (2002).  For 
values  in  the  designs  scaled  to  the  domain  [-1,  1]^  a  distance  list  is  defined 


composed  of  the  Euclidean  distance  between  all  pairs  of  design  points.  Mm 


v2y 


is  the  smallest  value  within  the  list.  The  Euclidean  distance  between  any  two 
design  points  (xi,  X2)  is  calculated  in  equation  2.5  (Morris  &  Mitchell,  1992).  As  a 
space-filling  measure,  a  larger  value  for  Mm  is  preferred. 


d(x„x,)= 


(2.5) 


/=i 


D.  ORTHOGONAL  LATIN  HYPERCUBE  (OLH)  DESIGNS 

The  construction  of  an  nxk  OLH  is  detailed  by  Ye  (1998).  The 
dimensions  of  n  and  k  can  be  determined  from  m  (any  positive  integer).  The 
number  of  runs,  n,  is  given  by 2'” -hi;  the  number  of  variables,  k,  is  given  by 
2/77  -  2 .  OLH  designs  are  denoted  as  O" . 

The  creation  of  an  OLH  requires  the  construction  of  three  matrices,  M,  S 

and  T. 


1.  Construction  of  Matrix  M  for  the  OLH 

The  matrix  M,  with  dimensions  qxk ,  consists  of  permutations  of  the  first 
column,  a  vector  e  representing  the  positive  levels  of  a  variable.  Although  vector 
e  can  be  any  ordering  of  the  first  q  integers  (1,2,. ..,g),  where  q  =  2"'  \  an  OLH 
design  can  be  guaranteed  when  vector  e  is  the  ascending  ordering  of  integers. 
For  example,  when  m  =  4,  q  =  2"^'^  =8,  e  =  [1  234567  8]^. 


11 


Permutation  matrices,  Al  (created  for  L  =  X  2,...,  m-^)  are  used  to 
generate  the  other  columns  of  matrix  M.  Each  of  the  permutation  matrices  is 
generated  by  the  following  equation,  where  matrix  I  is  the  2  x  2  identity  matrix. 


and  matrix  R  = 


0 

1 


1 

0 


=/(g)...(x)/(x)^(x)...(x)f?^  where  (8)  is  the  Kronecker  product®  (2.6) 

m-1-L  L 


For  example,  with  m  =  4  and  L  =  1,  the  permutation  matrix  Ai  =  I0I0R  (as 
seen  in  Table  1 )  yields  the  resultant  column  of  Aie  =  [21436597]^. 


Table  1 . 
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A  permutation  matrix  Ai  under  Ye’s  cons 
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=  4. 


Additional  permutation  matrices  can  be  created  by  multiplying  any  pair  of 
Ai  and  Aj,  where  i  j.  In  Ye’s  algorithm,  the  columns  of  M  are  composed  of 
vector  e,  Aje  for  /e{1,  2,  ...,  m-l}  and  AAm-ie  for  /e{1,  2,  ...,  m-2}.  The 


algorithm  uses  m-2  of  the 
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2-way  combinations  of  the  set  of  {Al}. 
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Table  2.  A  permutation  matrix  A1A3  under  Ye’s  construction  for  m  =  4. 


®  Given  a  mxn  matrix  A  and  a  pxq  matrix  B,  the  Kronecker  product  A®B  would  be  a 
(mxp)x(nxp)  matrix  C,  with  each  element  given  by  Cap  =  aybid  where  a  =  p(i-1)+k  and  p  =  q(j-1)+l. 
(Source:  Weisstein,  Eric  W.  "Matrix  Direct  Product."  From  MathWorld-A  Wolfram  Web  Resource. 
http://mathworld.wolfram.com/MatrixDirectProduct.htmh. 
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For  example,  with  m  =  4,  the  first  addition  is  the  permutation  matrix  A1A3 
(as  seen  in  Table  2),  which  yields  the  resultant  column  of  AiAse  =  [7  8  5  6  3 
4  1  2]\ 


An  example  of  matrix  M  for  m  =  4,  n  =  1 7,  /c  =  6,  is  in  Table  3. 
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Table  3.  A  matrix  M  under  Ye’s  construction  for  m  =  4. 


2.  Construction  of  Matrix  S  for  the  OLH 

Matrix  S  has  entries  ±1,  and  has  the  same  dimensions  as  matrix  M,  q  x  k. 
The  first  column  of  matrix  S  consists  of  +Ts,  and  the  subsequent  m-^  columns 
are  identical  to  a  two-level  full  factorial  design  for  m-^  variables.  The  remaining 
m-2  columns  are  created  by  pairwise  combinations  in  the  two-level  full  factorial, 
corresponding  to  the  interactions  between  Al  used  in  the  creation  of  matrix  M, 
i.e.,  A1A3.  For  example,  the  resultant  column,  i.e.,  S1S3,  is  obtained  by 
multiplying,  element  by  element,  the  columns  of  Si  and  S3  together. 


Ye  proposed  the  creation  of  the  matrix  S  by  defining  the  vector  aj  as 


■-T 

■f 

dj  =S^0B2  0...0B^_^, 

where 

Bj  = 

1  j 

and 

B;  = 

1 

j  e{1,  2,  ...,  m-^}  and  i;^  j.  This  vector  replaces  the  columns  of  Si,  ..,  Sm-i.  The 
subsequent  columns  are  pairwise  combinations  of  the  original  columns. 


For  example,  with  m  =  4,  the  vector  ai 
column  of  Si  =  [ -1  +1  -1  +1  -1  +1  -1  +1]^. 


-1 

1 


generates  the 
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An  example  of  the  matrix  S  for  m  =  4,  n  =  1 7,  /c  =  6,  is  in  Table  4. 


So 

Si 

S2 

S3 

S1S3 

S2S3 

+  1 

-1 

-1 

-1 

+  1 

+  1 

+  1 

+  1 

-1 

-1 

-1 

+  1 

+  1 

-1 

+  1 

-1 

+  1 

-1 

+  1 

+  1 

+  1 

-1 

-1 

-1 

+  1 

-1 

-1 

+  1 

-1 

-1 

+  1 

+  1 

-1 

+  1 

+  1 

-1 

+  1 

-1 

+  1 

+  1 

-1 

+  1 

+  1 

+  1 

+  1 

+  1 

+  1 

+  1 

Table  4.  A  matrix  S  under  Ye’s  construction  for  m  =  4. 


3.  Construction  of  Matrix  T  for  the  OLH 

Matrix  T,  of  dimensions  nxk,  is  created  by  the  Hadamard  product^  of 
matrix  M  and  matrix  S.  A  row  of  O’s  and  a  mirror  image  of  matrix  T  are  appended 
to  matrix  T  for  the  creation  of  an  OLH.  The  mirror  for  matrix  T  involves  changing 
the  sign  and  flipping  the  order  of  the  original  matrix  T. 


The  resultant  OLH,  O^,  with  e  =  [  1  2  3  4  5  6  7  8  ]^,  is  in  Table  5. 


Run 

Var  1 

Var  2 

Var  3 

Var  4 

Var  5 

Var  6 

1 

1 

-2 

-4 

-8 

7 

5 

2 

2 

1 

-3 

-7 

-8 

6 

3 

3 

-4 

2 

-6 

5 

-  7 

4 

4 

3 

1 

-5 

-6 

-  8 

5 

5 

-6 

-8 

4 

-3 

-  1 

6 

6 

5 

-7 

3 

4 

-2 

7 

7 

-8 

6 

2 

- 1 

3 

8 

8 

7 

5 

1 

2 

4 

9 

0 

0 

0 

0 

0 

0 

10 

-  8 

-  7 

-  5 

-  1 

-2 

-4 

11 

-  7 

8 

-6 

-2 

1 

-3 

12 

-6 

-5 

7 

-3 

-4 

2 

13 

-  5 

6 

8 

-4 

3 

1 

14 

-4 

-3 

-  1 

5 

6 

8 

15 

-  3 

4 

-2 

6 

-5 

7 

16 

-2 

- 1 

3 

7 

8 

-6 

17 

-  1 

2 

4 

8 

-  7 

-  5 

Table  5.  Og^  under  Ye’s  construction  (where  m  =  4,  k  =  6,  n  =  M). 


Based  on  the  measures  defined,  we  see  that  the  properties  for  this  Og^are 
cond{X^X)=  1 .0,  =  0.0,  Mm  =  1 .4737,  ML2  =  0.0965  and  CL2  =  0.2053. 


^  Suppose  A  =  (ay)  and  B  =  (by)  are  lo  n  x  m  matrices,  the  Hadamard  product  of  A  and  B  is 
the  entry-wise  product  of  A  and  B,  that  is  that  resultant  n  x  m  matrix  has  (i,j)  entries  of  aipij 
(http://planetmath.org/encyclopedia/HadamardProduct.html). 
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E.  NEARLY  ORTHOGONAL  LATIN  HYPERCUBE  (NOLH)  DESIGNS 

Cioppa  describes  the  construction  of  nx/c  OLH  and  NOLH  designs  by 
extending  Ye’s  algorithm  (Cioppa,  2002),  increasing  k  from  2m -2  to 

m  +  f  ,  with  the  number  of  runs,  n  ,  remaining  constant.  NOLH  designs  are 


denoted  as  N" . 


The  matrix  M  is  extended  by  using  all  possible  two-way  combinations  and 
appending  the  additional  columns,  AAje  for  i,j  =  X  2,  ...,  m-l  and  i j .  For 
example,  for  the  case  where  m  =  4,  the  extension  appends  the  columns,  {AiA2e, 
AiAse,  A2A3e}  instead  of  only  {AiA2e,  A2A3e}.  This  increases  the  number  of 
variables  explored,  as  depicted  in  Table  6.  Construction  of  matrix  S  then 
requires  the  same  number  of  additional  columns,  while  the  construction  for  matrix 
T  remains  unchanged. 


m 

n 

k  (Ye) 

k  (Cioppa) 

4 

17 

6 

7 

5 

33 

8 

11 

6 

65 

10 

16 

7 

129 

12 

22 

8 

257 

14 

29 

Table  6.  Comparison  for  k  between  Ye  and  Cioppa. 


By  varying  the  order  of  e  in  the  creation  of  the  matrix  M,  different  design 
matrices,  which  yield  different  properties,  can  be  constructed.  The  original  vector 
e  =  [1  234567  8]^  provides  an  orthogonal  design,  but  tends  to  have  poor 
space-filling  properties.  There  are  a  total  of  (2'”"')!  permutations  to  vector  e. 
This  creates  problems  of  searching  for  the  best,  as  seen  in  the  large  numbers  in 
Table  7. 


m 

Number  of  Permutations 

4 

40320 

5 

20922789888000 

6 

263130836933694000000000000000000000 

Table  7.  Number  of  possible  permutations  of  vector  e. 
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The  8-step  construction  proposed  by  Cioppa  is  detailed  (Cioppa  &  Lucas, 
2006)  as  follows: 

Step  1 :  Determine  the  number  of  variables  {k  >  1)  required.  If  the  number  of 

f  m-i\ 


variables  is  other  than  11,  1 6,  22,  or,  more  generally,  ( m  + 


V  2  , 


),  round  the 


required  number  of  variables  up  to  the  nearest  one  of  these  numbers. 


Step  2:  Establish  a  maximum  threshold  pairwise  correlation  value  and  a 
maximum  threshold  condition  number.  Based  on  extensive  experimentation,  we 
use  =  0.05,  0.17,  0.16,  and  COnd{X^X)  =  1.15,  2.4,  2.8  for  /c  =  11,  16, 
and  22,  respectively. 

Step  3:  Using  a  randomly  permuted  e,  construct  a  design  matrix. 

Step  4:  Calculate  the  pairwise  correlations  and  the  condition  number  of  the 
candidate  matrix. 


Step  5:  If  either  value  in  Step  4  exceeds  the  thresholds  in  Step  2,  discard  the 
design  and  return  to  Step  3  to  regenerate  with  another  randomly  permuted  e. 
Otherwise,  keep  the  design  and  proceed  to  Step  6.  Repeat  Steps  3-5  until  a 
desired  number  of  candidate  designs  are  found.  Note:  If  not  enough  are  found, 
relax  the  criteria  in  Step  2  and  begin  again.  We  have  found  that  15  candidate 
designs  work  well  for  Steps  6-8. 

Step  6:  Subject  each  of  the  candidate  designs  to  repeated  applications  of 
Florian’s  method  to  decrease  the  maximum  pairwise  correlation  and  condition 
number.  Stop  when  no  further  improvement  is  made. 


Step  7:  Calculate  the  Mm  distance  and  ML2  discrepancy  for  each  of  the  Step  6 
designs.  Rank  the  designs  according  to  some  combination  of  these  measures. 
We  chose  the  design  with  the  minimum  rank  sum  over  the  two  measures. 


Step  8:  If  a  number  of  variables  other  than  7,  1 1 ,  1 6,  22,  or  m  -i- 


m-^ 


IS 


required,  construct  each  possible  subset  having  the  appropriate  number  of 
columns  from  the  Step  7  design  and  calculate  the  Mm  distance  and  ML2 
discrepancy.  Choose  the  design  with  the  best  combination  of  these  two 
measures. 


With  preset  thresholds  of  cond{X^ X)<^.^3  and  yC'^g^<0.03,  Cioppa 
identifies  15  designs  that  satisfy  the  definition  of  near  orthogonality.  The  one 
with  the  best  space-filling  property  is  chosen  (as  in  Step  7)  and  catalogued  as  the 
best  NOLH  design.  A/" ,  for  the  particular  n  and  k  values. 
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F.  EXTENDING  OLH  AND  NOLH  DESIGNS 

With  Cioppa  extending  Ye’s  algorithm  to  include  all  2-way  combinations 
and  still  maintaining  orthogonal  properties,  the  exploration  into  the  inclusion  of  3- 
way  and  larger  combinations  for  the  permutation  matrices  for  construction  of  the 
LH  designs  was  performed.  Through  the  analysis,  it  is  confirmed  that  the 
orthogonal  properties  are  maintained,  thus  allowing  the  ability  to  investigate  a 
larger  number  of  factors  with  the  same  number  of  design  points  in  an  OLH 
design. 

For  each  design  matrix,  the  inclusion  of  p-way  combinations  also  includes 
all  smaller  combinations  for  the  permutation  matrices.  For  example,  an  inclusion 
of  a  4-way  combination  also  includes  both  the  2-way  and  3-way  combinations. 
The  number  of  factors,  k,  with  the  inclusion  of  p-way  combinations  (where  the 
maximum  is  p  <  m  - 1 ),  is  given  by  the  following  equation: 


V  j  y 


(2.7) 


The  number  of  factors  for  each  design  is  improved  as  shown  in  Table  8. 


n 

m 

Ye’s 

OLH 

Cioppa 

(2-way) 

Ang 

3-way 

4-way 

p-way^ 

9 

3 

4 

4 

- 

- 

4 

17 

4 

6 

7 

8 

- 

8 

33 

5 

8 

11 

15 

16 

16 

65 

6 

10 

16 

26 

31 

32 

129 

7 

12 

22 

42 

57 

64 

257 

8 

14 

29 

64 

99 

128 

513 

9 

16 

37 

93 

163 

256 

1025 

10 

18 

46 

130 

256 

512 

Table  8.  Comparison  of  LH  designs  with  p-way  combinations. 


The  construction  of  the  LH  designs  remains  similar  to  the  methods 
proposed  by  Ye,  and  also  by  Cioppa.  The  additional  permutation  matrices 
include  all  the  combinations  of  A  A  ...A  for  x,,x,,..,x  e  1,2,...,m-1 ,  and 

X2  Xp  I  z  p 

7^  X2  ^  Xp.  For  example,  for  p  =  3,  the  2-way  combinations  AjAj,  for  i  ^  j , 


®  The  maximum  possible  p-way  combination  depends  on  the  value  of  m,  where  p  =  m  -1 . 
For  example,  if  m  =  4,  the  maximum  p-way  combination  is  a  3-way  combination. 
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and  3-way  combination  AjAjAk  are  included  for  the  permutation  matrices,  for  all 
i,  j,  k  e  1 ,2,...,m-1  and  \  ^  j  . 

For  m  =  4,  the  extension  appends  the  column,  {AiA2A3e}  in  addition  to  the 
2-way  combinations.  Construction  of  matrix  S  includes  the  additional  number  of 
columns  in  a  similar  fashion.  The  construction  of  matrix  T  remains  unchanged. 
The  details  for  the  construction  of  the  design  matrix  Og^  can  be  seen  in  Appendix 
A. 

With  the  inclusion  of  additional  combinations  for  more  factors,  there  are 
tradeoffs  in  the  orthogonal  and  space-filling  properties  for  the  individual  designs. 
For  orthogonal  properties,  the  extended  has  the  same  properties  as  the 

methods  by  Ye  and  Cioppa.  The  space-filling  properties  for  the  designs  with 
larger  numbers  of  combinations  will  be  worse  when  compared  with  the  other 
designs,  because  we  are  putting  the  same  number  of  design  points  in  a  larger 
space.  A  table  of  comparison  is  shown  in  Table  9. 


cond{X^X) 

Pamp 

Mm 

ML2 

CL2 

Ye  07 

1.0 

0.0 

1.4737 

0.0965 

0.2053 

Cioppa  07 

1.0 

0.0 

1.479 

0.1732 

0.2551 

Ang  0^' 

1.0 

0.0 

1.7854 

0.3144 

0.3142 

Table  9.  Properties  of  O^,  O^and  O'^ . 


For  designs  with  a  fixed  number  of  runs,  n,  there  are  usually  tradeoffs  for 
increasing  the  number  of  variables,  k,  for  the  design.  The  increase  in  k, 
increases  the  dimensionality  of  the  design  matrix,  and  improves  Mm.  Flowever, 
the  ML2  and  CL2  measures  reflect  worse  space-filling  properties  for  a  larger  k. 
For  NOLFI  designs,  the  limits  used  are  cond{X^X)  <  1 .2  and  <  0.05 . 
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III.  DISCRETE/CATEGORICAL  VARIABLES  AND  THEIR 
EFFECT  ON  PROPERTIES  OF  LH  DESIGNS 

A.  INTRODUCTION 

A  discrete  variable  is  a  variable  that  assumes  values  within  a  discrete  set, 
such  as  integers.  Discrete  variables  can  either  be  nominal  or  ordinal  variables. 
Nominal  variables  are  variables  where  order  does  not  matter,  such  as  gender. 
Ordinal  discrete  variables  have  an  ordering  or  quantitative  value  associated,  e.g., 
the  number  of  tanks  and  military  ranks. 

An  example  of  a  nominal  discrete  variable  is  gender.  Male  and  Female 
are  often  represented  in  designs  in  the  continuous  range  by  values  1  and  0,  or  0 
and  1 ,  respectively,  as  there  is  no  correlation  between  the  representation  and  the 
nominal  variable.  Another  example  of  a  nominal  discrete  variable  with  more  than 
two  levels  is  the  military  service,  which  include  Army,  Navy,  and  Air  Force.  In 
general,  nominal  discrete  variables  with  a  levels  require  a-\  indicator 
variables. 

An  example  of  an  ordinal  discrete  variable  is  the  number  of  soldiers  or 
tanks,  which  can  take  only  integer  values.  Another  example  of  an  ordinal 
discrete  variable  is  the  military  ranks  for  the  Singapore  Army,  which  have  an 
ordinal  scale. 

Our  investigation  of  the  effects  of  discrete  variables  on  the  properties  of 
the  LFI  designs  seeks  to  answer  the  following  questions: 

•  Flow  does  the  choice  of  the  column  used  for  the  discrete  variable 
affect  the  properties  of  the  LFI  design? 

•  What  are  the  effects  on  the  properties  of  the  LFI  design  for 
increasing  numbers  of  discrete  variables? 

•  Flow  do  different  levels  for  discrete  variables  affect  the  properties  of 
the  LFI  design? 
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•  How  does  different  coding  of  discrete  variables  affect  the  properties 
of  the  LH  design? 


B.  METHODOLOGY  OF  EVALUATION 

The  methodology  used  to  investigate  the  effects  of  discrete  variables  on 
the  properties  of  LH  designs  looks  at  the  effects  of  representing,  coding  and 
ordering  of  discrete  variables.  OLH  and  NOLH  design  matrices  constructed 
earlier  are  used,  and  the  results  of  this  evaluation  are  discussed  in  Chapter  IV. 

1.  Representing  Types  of  Discrete  Variables 

Both  nominal  and  ordinal  discrete  variables  are  investigated.  Ordinal 
discrete  variables  are  more  significant,  as  the  presence  of  ordinal  variables  is 
extremely  common  in  real  world  problems.  This  thesis  looks  at  the  coding 
methods  for  both  nominal  and  ordinal  discrete  variables  in  the  following  sections. 

2.  Coding  Discrete  Variables 

Coding  of  nominal  discrete  variables  with  {high,  low}  can  be  done  in  (at 
least)  two  different  methods,  because  the  ordering  of  the  values  does  not  matter. 
The  two  methods  we  look  at  are  positive-negative  coding  and  odd-even  coding. 
Positive-negative  coding  assigns  all  positive  values  in  our  earlier  LH  designs  with 
the  high  value  and  negative  values  with  the  low  value.  Odd-even  coding  assigns 
all  even  numbers  with  the  high  value  and  the  odd  numbers  with  the  low  value. 


For  example,  both  coding  for  the  design  are  shown  in  Table  10. 


Original 

-8 

-7 

-6 

-5 

-4 

-3 

-2 

-1 

0 

1 

2 

3 

4 

5 

6 

7 

8 

Positive- 

Negative 

lo 

lo 

lo 

Lo 

lo 

lo 

lo 

lo 

hi 

hi 

hi 

hi 

hi 

hi 

hi 

hi 

hi 

Odd- 

Even 

hi 

lo 

hi 

Lo 

hi 

lo 

hi 

lo 

hi 

lo 

hi 

lo 

hi 

lo 

hi 

lo 

hi 

Table  10.  Coding  methods  for  nomina 


variables. 
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The  mathematical  formulation  of  the  positive-negative  (PN)  and  odd-even 
OD)  coding  are: 


codingp^ 


Jmax  value  if  x>0 
[min  value  if  x<0 


remainder  =  MOD{value,2) 

j max  value  if  remainder  =  0 
1  min  value  if  remainder  0 


coding^p  = 


Coding  discrete  variables  with  more  than  2  levels  is  done  by  distributing 
the  levels  uniformly  across  the  number  of  levels  of  the  discrete  variables.  For 
example,  the  coding  for  the  3-level  ordinal  discrete  variable  {0,  1,  2}  in  the 
design  is  represented  in  Table  11. 


Original 

-8 

-7 

-6 

-5 

-4 

-3 

-2 

-1 

0 

1 

2 

3 

4 

5 

6 

7 

8 

Coding 

0 

0 

0 

0 

0 

0 

1 

1 

1 

1 

1 

2 

2 

2 

2 

2 

2 

Table  1 1 .  Coding  for  ordinal  variables. 


3.  Ordering  Columns  of  a  LH  Design  for  Additional  Discrete 
Variables 

For  a  single  discrete  variable,  the  best  column  to  represent  the  discrete 
variable  is  obtained  after  comparing  the  orthogonal  and  space-filling  properties  of 
the  design  with  the  discrete  variable  coded.  All  of  the  columns  are  used  for  the 
single  discrete  variable  to  generate  additional  designs.  The  properties  for  each 
new  design  are  ranked,  and  the  column  which  gives  the  best  properties  should 
be  used  to  represent  the  single  discrete  variable. 

For  example,  the  design  is  used  to  represent  a  single  2-level  discrete 
variable.  After  generating  7  new  designs,  the  properties  of  the  new  designs  are 
collected  and  ranked,  as  seen  in  Table  12.  The  best  of  the  new  designs  (with  the 
lowest  overall  rank)  is  the  choice  of  the  column  for  the  2-level  discrete  variable. 
Column  5  is  the  best  choice  for  to  represent  the  2-level  discrete  variable. 
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Col. 

Used 

Raw  results 

Ranking 

cond{X^X) 

Pamp 

Mm 

ML2 

cond{X^X) 

P amp 

Mm 

MLz 

Overall 

Rank 

1 

4.004 

0.385 

1.431 

0.627 

4 

2 

3 

1 

10 

2 

4.004 

0.385 

1.392 

0.629 

4 

2 

4 

2 

12 

3 

4.004 

0.385 

1.458 

0.640 

4 

2 

2 

4 

12 

4 

4.004 

0.385 

1.275 

0.665 

4 

2 

6 

7 

19 

5 

2.946 

0.193 

1.474 

0.631 

1 

1 

1 

3 

5  (Best) 

6 

3.710 

0.385 

1.341 

0.648 

2 

2 

5 

6 

15 

7 

3.928 

0.385 

1.192 

0.645 

3 

2 

7 

5 

17 

Table  12.  Ranking  and  selection  of  columns  for  discrete  variab 


es. 


The  selection  of  columns  for  new  LH  designs  for  multiple  discrete 
variables  will  choose  the  best  column,  and  sequentially  choose  the  other 
columns  that  give  the  best  orthogonal  and  space-filling  properties.  An  iterative 
process  determines  the  columns,  and  although  this  process  may  not  provide  the 
best  (or  global  optimal)  result,  the  saving  of  computational  time  promotes  its  use 
over  the  total  enumeration  of  all  possibilities. 

The  ordering  of  columns  to  use  for  the  designs,  for  additional  2-level 
variable  columns  for  the  design,  is  {5,  2,  6,  3,  7,  4,  1}.  An  example  of 
interpreting  this  result  for  a  scenario  with  two  2-level  discrete  variables  using  the 
07  design  is  that  we  should  use  columns  5  and  2,  because  they  provide  the  best 

orthogonal  and  space-filling  properties.  This  is  further  described  with  illustration 
in  Chapter  IV,  Tables  19  and  20. 
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IV.  RESULTS  OF  RESEARCH 


A.  SOFTWARE  IMPLEMENTATION  OF  LATIN  HYPERCUBE 
GENERATOR  IN  JAVA 

Within  the  IDE^,  running  primarily  on  a  personal  laptop'll  or  lab 
computer'''',  OLH  and  NOLH  designs  corresponding  to  Table  13  were 
successfully  generated.  Properties  of  the  individual  designs  are  discussed  in 
section  B.  The  largest  OLH  design,  using  e  =  [  1  2  ...  512  took 

approximately  30  minutes  to  generate.  The  time  taken  to  generate  different 
NOLH  designs  is  largely  dependent  on  the  number  of  candidate  designs 
considered. 


n 

m 

Additional  p-way  combinations  for  the  permutation  matrix 

2-way 

3-way 

4-way 

5-way 

6-way 

7 -way 

8 -way 

9 -way 

17 

4 

7 

8 

NA 

NA 

NA 

NA 

NA 

NA 

33 

5 

11 

15 

16 

NA 

NA 

NA 

NA 

NA 

65 

6 

16 

26 

31 

32 

NA 

NA 

NA 

NA 

129 

7 

22 

42 

57 

63 

64 

NA 

NA 

NA 

257 

8 

29 

64 

99 

120 

127 

128 

NA 

NA 

513 

9 

37 

93 

163 

219 

247 

255 

256 

NA 

1025 

10 

46 

130 

256 

382 

466 

502 

511 

512 

Table  13.  List  of  LH  designs  generated. 


Inclusion  of  p-way  combinations  of  the  permutation  matrices  during 
construction  of  OLH  and  NOLH  designs  can  achieve  a  larger  LH  design.  The 
catalogue  of  OLH  designs  captures  all  designs,  and  the  catalogue  of  NOLH 
designs  is  still  in  progress.  The  catalogues  can  be  found  on  the  SEED  Center 
website  (http://harvest.nps.edu):  some  of  the  designs  are  catalogued  in  Appendix 
B. 

The  code  for  the  generation  of  OLH  and  NOLH  is  also  available  on  the 
SEED  Center  website. 


9  Eclipse  SDK  (Version:  3.2.0)  Build  id:  M20060629-1905  (Available  via: 
http://wvvw.eclipse.orq/downloads/).  Last  date  accessed,  Nov  2006 

''^  Home  laptop  -  Fujitsu  running  Intel®  Pentium®  M  1 .73  GHz,  1 .00GB  RAM. 

''''  Lab  computer  -  Dell  running  Intel®  Pentium®  4  3.40  GHz,  2.00  GB  RAM. 
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Construction  of  NOLH  designs  are  done  in  two  ways,  with  varying 
performances  in  speed  and  properties  of  LH  designs,  namely  random  and 
sequential.  The  random  permutation  of  the  vector  e,  generates  NOLH  design  in 
slightly  longer  time,  but  with  good  properties.  The  sequential  method  searches 
around  a  known  OLH  design  and  generates  the  NOLH  design  much  faster,  but 
its  properties  are  dependent  on  the  initial  starting  point.  An  illustration  of  the  idea 
for  both  methods  is  shown  in  Figure  3. 


©  Candidate  Design  from 
random  permutated  e  vector 

0  Candidate  Design  from  fixed 
permutated  e  vector  about  a 
known  OLH 

O  Known  OLH 


Figure  3.  Illustration  of  the  random  and  fixed  permutation  of  the  e  vector  for 
construction  of  the  NOLH  designs  (Best  viewed  in  color). 

1.  Summary  of  Functions  within  JAVA  Application 

The  JAVA  application  is  able  to  do  the  following: 

•  Construct  OLH  to  0|  to  Og^^ . 

•  Construct  NOLH  fromA/|to  A/|3^(work  is  still  ongoing  as  “is 
forever”). 

•  Read  in  LH  design  (in  .txt  format)  and  output  the  orthogonal  and 
space-filling  properties  of  the  design. 
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•  Read  in  LH  design  (in  .txt  format)  and  perform  a  categorical 
analysis  on  the  design  matrix,  and  output  the  order  of  the  columns  to 
use  which  give  good  properties  to  the  now,  mixed  design. 

•  Read  in  LH  design  (in  .txt  format)  and  stack  the  design  based  on 
user’s  input,  and  output  the  design  and  its  corresponding  orthogonal 
and  space-filling  properties. 

2.  Catalogues  of  OLH  and  NOLH  Designs 

A  partial  catalogue  of  OLH  designs  is  included  in  Appendix  B.  Current 
designs  include  the  construction  for  m  <  10  (up  to  and  are  available  on  the 
SEED  Center  website. 

A  catalogue  of  NOLH  designs,  where  the  thresholds  are  cond{X^X)  <1.2 
and  <  0.05 ,  is  available  on  the  SEED  Center  website.  Construction  of 

NOLH  designs  will  continue  to  obtain  better  design  matrices,  and  updated  design 
matrices  will  also  be  available. 

With  the  ability  to  extend  the  OLH  and  NOLH  designs  by  the  inclusion  of 
additional  p-way  combinations  of  permutation  matrices,  the  design  matrix  of  O®^  is 
in  Table  14. 
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B.  PROPERTIES  OF  OLH  AND  NOLH  DESIGNS 


1.  Experimenting  and  Extending  the  OLH  and  NOLH  Designs 

Through  experimentation,  new  design  matrices  to  include  up  to  p-way 
combinations  of  permutation  matrices  were  constructed.  These  designs  have 
maintained  their  orthogonal  properties  of  cond{X^ X)=^.0  and  =  0.0,  even 

with  the  inclusion  of  3-way,  4-way,  ...,  (/77-1)-way  combinations  when  the  vector 
e  is  [1,  2,  ...,  2'^'^]^.  The  properties  for  the  LH  designs  that  include  3-way  and  4- 
way  combinations  are  in  Table  15,  in  comparison  with  the  mean  of  100  Flexible 
Random  Latin  Hypercube  (FRLHS)  designs. 


m 

n 

3-way 

4-way 

Mean  of  100  FRLHS 

k 

cond{X^X) 

Pamp 

Mm 

MLz 

k 

cond{X^  X) 

Pamp 

Mm 

MLz 

k 

cond{X^  X) 

Pamp 

Mm 

MLz 

4 

17 

8 

1.0 

0.0 

1.78 

0.314 

8 

NA 

NA 

8 

2.47 

0.24 

1.27 

0.35 

5 

33 

15 

1.0 

0.0 

2.20 

7.01 

16 

1.0 

0.0 

2.42 

11.52 

15 

2.11 

0.17 

1.96 

6.23 

6 

65 

26 

1.0 

0.0 

2.70 

8.67E2 

31 

1.0 

0.0 

3.18 

1.04E4 

26 

1.77 

0.10 

2.77 
4.31  E2 

7 

129 

42 

1.0 

0.0 

3.27 

1.76E6 

57 

1.0 

0.0 

4.14 

5.24E9 

42 

1.54 

0.07 

3.27 

1.78E5 

Table  15.  Construction  of  OLH  designs  with  3-way  and  4-way  combinations. 


Orthogonal  properties  of  the  LH  designs  are  maintained  with  the  inclusion 
of  higher  order  combinations,  but  these  designs  have  less  ideal  space-filling 
properties.  Depending  on  the  type  of  scenario  and  the  limitations  to  the  number 
of  runs,  different  design  matrices  may  be  used. 

After  the  initial  test  with  additional  columns  confirmed  success  in 
generating  OLH  designs,  additional  OLH  designs  were  generated  and  the 
properties  of  the  designs  are  shown  in  Table  16. 

Observations  for  these  designs  include: 

a)  Increasing  the  number  of  possible  combinations  of  the  permutation 
matrices  increases  the  number  of  factors  for  the  design  matrices. 
However,  the  space-filling  nature  is  less  favorable  in  comparison  with 
designs  with  fewer  combinations  of  the  permutation  matrices. 


27 


k 

n 

cond(X^X) 

P amp 

Mm 

ML2 

CL2 

7 

17 

1.00 

0.00 

1.48 

0.17 

0.26 

8 

17 

1.00 

0.00 

1.79 

0.31 

0.31 

11 

33 

1.00 

0.00 

1.67 

0.95 

0.41 

15 

33 

1.00 

0.00 

2.20 

7.01 

0.73 

16 

33 

1.00 

0.00 

2.42 

11.52 

0.83 

16 

65 

1.00 

0.00 

1.79 

7.98 

0.72 

22 

129 

1.00 

0.00 

1.79 

96.58 

1.39 

26 

65 

1.00 

0.00 

2.70 

867.17 

2.25 

29 

257 

1.00 

0.00 

1.81 

1701.28 

2.92 

31 

65 

1.00 

0.00 

3.19 

1.E+04 

3.816 

32 

65 

1.00 

0.00 

3.34 

2.E+04 

4.261 

37 

513 

1.00 

0.00 

1.86 

5.E+04 

6.81 

42 

129 

1.00 

0.00 

3.27 

2.E+06 

11.34 

46 

1025 

1.00 

0.00 

1.91 

2.E+06 

18.24 

57 

129 

1.00 

0.00 

4.14 

5.E+09 

49.27 

Table  16.  Properties  of  OLH  designs  (from  to  O™). 


b)  The  results  shown  are  intuitive.  For  example,  comparing  the 
designs  and  0“ ,  shows  that  a  design  with  more  runs,  n,  improves  the 
space-filling  property  of  the  design  for  a  fixed  number  of  variables. 

c)  The  orthogonal  properties  of  an  OLH  design  (where  cond(x^x)  =  1 .0, 
=  0.0)  are  dependent  on  the  permutation  of  vector  e.  With  a  total 

enumeration  of  the  possible  permutations  of  vector  e  for  a  17  run  design, 
there  are  a  total  of  640  O^and  640  Og^  These  designs  result  from  the 

same  vector  e  that  is  used  in  the  construction.  The  total  enumeration  of 
larger  designs  is  not  possible  because  the  total  number  of  permutations  of 
the  vector  e  becomes  too  big,  as  shown  earlier  in  Table  7. 

Comparing  these  640  designs,  the  properties  for  the  best  space¬ 
filling  designs  are  in  Table  17.  The  best  designs  show  improvements  to 
the  space-filling  property,  ML2.  A  pairwise  plot  of  the  best  Og^  design  is 
shown  in  Figure  4. 
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k 

n 

cond{X^X) 

P amp 

Mm 

ML2 

CL2 

Remarks 

7 

17 

1.00 

0.00 

1.48 

0.17 

0.26 

e  =  [1,2,..,8] 

7 

17 

1.00 

0.00 

1.479 

0.1519 

0.2426 

Best  of  640 

8 

17 

1.00 

0.00 

1.79 

0.31 

0.31 

e  =  [1,2,..,8] 

8 

17 

1.00 

0.00 

1 .7854 

0.2893 

0.3003 

Best  of  640 

Table  17.  Properties  of  and  Og^  with  total  enumeration  of  e. 
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d)  The  extension  of  Ye’s  algorithm  maintains  the  following  two 
theorems  (Ye,  1998): 


Theorem  4  (Ye,  1998).  For  the  LH  design  constructed, 

the  elementwise  square  of  each  column  is  orthogonal  to  all  the 
columns  in  the  LH  design. 

Theorems  5  (Ye,  1998).  For  the  LH  designs  constructed, 
the  elementwise  product  of  every  two  columns  is  orthogonal  to 
all  columns  in  the  LH  design. 

The  result  of  the  two  theorems  provides  subsequent  estimates  of 
quadratic  and  bilinear  interaction  effects  that  are  uncorrelated  with  the 
estimates  of  the  linear  effects. 

2.  Exploring  the  Construction  of  Latin  Hypercube  Designs 

OLH  and  NOLH  designs  can  be  constructed  by  different  methods,  and  two 
methods  of  construction  were  compared  against  catalogued  designs.  The  two 
methods  include  random  generation  and  using  algorithms  proposed  by  Ye  and 
Cioppa.  The  driving  factors  for  this  investigation  are  the  speed  and  ease  of 
construction  vis-a-vis  the  orthogonal  and  space-filling  properties  of  the  LH 
designs. 

From  the  100  designs  generated  from  Random  Latin  Hypercube  Sampling 
(RLHS)''2,  Florian’s  method  was  performed  on  the  RLHS  designs,  obtaining  new 
designs  called  Flexible  RLHS  (FRLHS)  designs  with  improvements  to  the 
orthogonal  properties.  The  mean  of  these  100  designs  are  used  for  the 
comparison. 

Designs  that  were  constructed  from  the  algorithm  include  OLH  and  NOLH 
designs  and  the  existing  catalogue  of  designs  from  the  SEED  Center  website. 


RLHS  method  used  MATLAB’s  X  =  lhsdesign(n,p)  that  generates  a  Latin  Hypercube 
sample  X  containing  n  values  on  each  of  p  variables.  The  RLHS  refers  to  the  average  of  100 
RLHS  designs.  FRLHS  are  Flexible  RLHS  design  that  used  Florian’s  method  to  improve  the 
orthogonal  properties. 
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A  subset  of  the  comparison  of  the  properties  can  be  seen  in  Table  18,  with 
a  larger  compilation  of  results  seen  in  Appendix  C. 


K 

N 

Type 

condiX'^X) 

P amp 

Mm 

MLz 

OLH 

1.00 

0.00 

1.48 

0.17 

SEED 

1.00 

0.00 

1.48 

0.15 

7 

17 

NOLH 

1.17 

0.05 

1.44 

0.16 

RLHS 

9.53 

0.52 

0.97 

0.57 

FRLHS 

2.24 

0.23 

1.11 

0.19 

OLH 

1.00 

0.00 

1.79 

0.31 

SEED 

NA 

NA 

NA 

NA 

8 

17 

NOLH 

1.06 

0.03 

1.79 

0.30 

RLHS 

14.41 

0.57 

1.12 

0.92 

FRLHS 

2.47 

0.24 

1.27 

0.35 

Table  18.  Properties  of  LH  designs  of  different  construction  (for  n  =  17). 


Four  properties  were  considered  while  comparing  the  different  LH 
designs:  they  are  the  orthogonal  properties  cond{X^X)  and  ;  and  the  space¬ 
filling  properties  Mm  and  ML2. 

The  comparison  of  the  properties  is  based  on  the  results  for  50  different 
designs,  using  11  designs  (each)  from  the  RLHS,  FRLHS,  OLH  and  NOLH 
designs;  and  6  designs  from  the  NOLHdesign_v4.xls  (available  on  the  SEED 
Center  website). 

In  comparing  the  speed  and  the  ease  of  construction,  the  RLHS  designs 
and  the  NOLH  designs  from  SEED  are  the  easiest  and  fastest,  followed  by  the 
FRLHS  designs,  OLH  designs,  and  NOLH  designs. 

A  parallel  plot  of  the  properties  for  the  different  LH  designs  based  on 
different  construction  methods  can  be  seen  in  Figure  5. 
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Figure  5.  Comparison  of  speed  and  ease  of  construction  vs.  properties  of  LH 

designs  (Best  viewed  in  color). 

The  parallel  plot  in  Figure  5  reflects  the  improvement  of  Florian’s  method 
on  the  RLFIS  to  RLFIS-F  designs,  as  seen  in  charts  (b)  and  (c).  The  odd  line  of 
the  OLFI  designs  (chart  (d))  is  due  to  poor  space  filling  properties  of  larger 
designs.  OLH  and  NOLH  designs  show  similar  space  filling  properties, 
comparable  to  the  average  of  100  RLHS  and  RLHS-F  designs.  The  notable 
differences  between  the  different  construction  methods  for  the  LH  designs  are 
that: 

a.  Florian’s  method  improves  the  orthogonal  properties  of  the  design 
matrices  of  the  RLHS,  and  the  improvement  is  largely  dependent  on  the  n 
and  k  values.  The  comparison  of  matrices  with  similar  n  and  k  reveals  that 
the  orthogonal  properties  are  slightly  worse  than  the  OLH  and  NOLH 
designs.  However,  RLHS  designs  have  attractive  space-filling  properties. 
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b.  For  smaller  design  matrices,  the  space-filling  properties  do  not 
differ  greatly  from  the  OLH  with  the  NOLH  or  RLHS.  However,  the  Mm 
and  ML2  measures  tend  to  reflect  a  downgraded  performance  with  larger 
design  matrices. 

c.  The  comparison  between  design  matrices  with  the  same  number  of 
runs,  n,  i.e.,  including  the  design  matrices  with  2-way,  3-way  and  4-way 
combinations  of  permutation  matrices,  reveals  that  the  inclusion  of  higher 
order  combinations  of  permutation  matrices  causes  downgraded 
properties  for  cond{X^X),  p^^p,  and  ML2,  but  an  improved  Mm  property, 

as  there  is  a  bigger  space  in  which  to  push  the  design  points  apart. 

3.  Exploring  the  OLH  and  NOLH  Designs  for  Discrete  Variables 

Exploration  into  the  representation  of  discrete  variables,  both  nominal  and 
ordinal,  is  important  to  fitting  a  scenario  into  a  chosen  design  matrix.  From  a 
user’s  point  of  view  of  a  NOLH  tool  for  the  design  of  experiments,  having  in-depth 
knowledge  on  the  method  of  construction  for  the  different  designs  for  different 
scenarios  may  not  be  important.  But,  given  a  choice,  the  user  should  prefer  a 
better  design  if  available. 

The  exploration  looked  into  the  modifications  of  an  LH  design  with  the 
inclusion  of  discrete  data,  to  find  the  “best”  design  from  the  original  LH  design. 

Interesting  results  were  obtained  from  the  study  into  the  properties  of  OLH 
and  NOLH  designs  with  different  numbers  and  levels  of  discrete  variables. 


a.  For  2-level  discrete  variables,  Odd-Even  coding  leads  to  problems 
with  increased  numbers  of  variables.  Although  the  coding  provides  for  a 
better  orthogonal  property  with  the  addition  of  a  single  discrete  variable 
with  2-levels,  it  leads  to  problems  when  additional  2-level  discrete 
variables  are  to  be  included  for  the  design.  The  comparison  between 
successive  additions  of  2-level  discrete  variables  on  O^with  different 
coding  methods  is  shown  on  Table  19. 
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For  example,  when  the  O^with  Odd-Even  coding  requires  three  2-level 
discrete  variables,  it  has  =1.  The  result  of  =1  is  also  reflected  for  all 

’  amp  amp 


other  OLH  and  NOLH  designs. 


Properties  of  07-17  with  Pos-Nea  Codina 

Pro 

perties  of  07-17  with  Odd-Even  Codina 

cond(X^X) 

Pamp 

Mm 

MLi 

cond{X^X) 

Pamp 

Mm 

MLi 

Oric 

final 

1.00 

0.00 

1.48 

0.17 

Oric 

final 

1.00 

0.00 

1.48 

0.17 

Order  of  Columns 
changed 

5 

2.946 

0.192 

1.474 

0.631 

Order  of  Columns 

changed 

3 

2.667 

0.000 

1.458 

0.622 

2 

3.817 

0.385 

1.244 

1.062 

1 

16.000 

0.889 

1.639 

1.165 

6 

3.042 

0.096 

1.237 

1.483 

6 

1.6E+17 

1.000 

1.620 

1.868 

3 

4.355 

0.385 

1.125 

1.912 

4 

3.4E+17 

1.000 

0.612 

2.492 

7 

3.613 

0.192 

1.008 

2.349 

5 

1.5E+49 

1.000 

0.559 

3.386 

4 

4.743 

0.385 

1.000 

2.840 

2 

Infinity 

1.000 

0.250 

4.028 

1 

1.438 

0.056 

0.000 

3.281 

7 

Infinity 

1.000 

0.000 

5.070 

Table  19.  Prob 

ems  with  Odd-Even  coding  with  addition  2-leve 

discrete 

variables  (demonstrated  on  0^  )■ 


The  07 design,  with  Positive-Negative  coding,  starts  with  cond{X^X)  =  i 
and  =  0  (as  seen  in  Table  19).  If  one  2-level  discrete  variable  is  required,  the 


use  of  column  5  gives  the  best  properties,  as  reflected  in  the  chart.  If  two  2-level 
discrete  variables  are  required,  the  use  of  column  5  and  column  2  results  in  a  LH 
design  with  better  orthogonal  and  space-filling  properties. 


Properties  of  07-17  with  Pos-Neq  Coding 

Properties  of  07-16  with  Pos-Neqative  Coding 

cond(X^X) 

Pamp 

Mm 

MLi 

condiX'^X) 

Pamp 

Mm 

MLi 

Oric 

final 

1.00 

0.00 

1.48 

0.17 

Oric 

final 

1.00 

0.00 

1.69 

0.23 

Order  of  Columns 
changed 

5 

2.946 

0.192 

1.474 

0.631 

Order  of  Columns 
changed 

5 

2.805 

0.198 

1.969 

0.669 

2 

3.817 

0.385 

1.244 

1.062 

2 

3.613 

0.396 

1.904 

1.086 

6 

3.042 

0.096 

1.237 

1.483 

6 

2.567 

0.099 

2.512 

1.466 

3 

4.355 

0.385 

1.125 

1.912 

1 

3.926 

0.396 

2.559 

1.832 

7 

3.613 

0.192 

1.008 

2.349 

4 

2.805 

0.198 

2.610 

2.173 

4 

4.743 

0.385 

1.000 

2.840 

3 

3.845 

0.396 

2.872 

2.494 

1 

1.438 

0.056 

0.000 

3.281 

7 

1.000 

0.000 

3.464 

2.789 

Table  20.  Pos-Neg  coding  with  additional  2-level  variables  for  and  O7®. 
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Removing  the  centre  row  of  zeros  for  obtains  the  O]^  design.  The 
effect  of  additional  2-level  discrete  variables  results  in  an  orthogonal  design  when 
all  the  columns  are  used  to  represent  2-level  discrete  variables  (as  seen  in  Table 
20).  This  result  holds  true  for  the  OLH  designs. 

b.  For  ordinal  variables,  the  number  of  levels,  /,  for  the  discrete 
variables  affects  the  performance  of  the  design  matrix.  Each  column  will 
only  have  values  from  1,2,  ..,  /  if  the  column  is  used  to  represent  an  /-level 
discrete  variable.  For  example,  if  representing  a  3-level  discrete  variable, 
the  column  used  in  will  be  [12312312312312312]^. 

The  properties  (in  pa/^icular,  the  orthogonal  measures)  of  are  affected 
by  fewer  levels,  and  the  results  stabilize  as  the  levels  increase.  Figure  6  shows 
the  properties  for  with  different  levels  of  discrete  variables,  similar  for  other 
OLFI  and  NOLH  designs. 


Figure  6.  Comparison  of  properties  when  one  /-level  discrete  variable  is  added  to 

the  design  (Best  viewed  in  color). 
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c.  There  is  no  effect  on  the  orthogonal  properties  for  the  addition  of 
discrete  variable  columns  for  “complete”''^  designs,  which  includes  , 

0^6’  ^32’  ■■■>  ^512^-  “Complete”  designs  do  not  have  variations  in  the 
orthogonal  properties  when  the  different  columns  are  selected,  unlike  the 
smaller  designs  which  result  in  variation  in  the  orthogonal  properties. 
However,  some  of  the  space-filling  properties  have  minor  changes. 
Though  with  poorer  space-filling  properties  compared  to  “not  complete” 
designs,  this  property  could  be  useful  to  note  while  considering  the  design 
to  use,  i.e.,  the  use  of  the  OLH  and  NOLH  designs  with  a  larger  number  of 
discrete  variables. 


4.  Exploring  the  Effects  of  Stacking  on  OLH  and  NOLH  Designs 

Stacking  with  random  permutation  of  the  columns  improves  the  space¬ 
filling  properties  of  LH  designs.  The  basic  principle  behind  stacking  is  that  the 
random  permutation  of  columns  maintains  the  orthogonal  properties  of  an  OLH 
design.  Adding  additional  copies  of  the  designs  with  random  permutation 
improves  the  space-filling  properties  while  maintaining  orthogonal  properties. 
Furthermore,  these  are  easily  constructed  and  allow  replication  at  each  level. 


The  notation  of  siO^  describes  a  design  with  an  appended  with  s-1 


other  with  randomly  permuted  columns.  For  example,  the  illustration  in 


Figure  7  shows  the  construction  of  2:02- 


1 

-2 

2 

1 

0 

0 

-1 

2 

-2 

-1 

1 

-2 

2 

1 

-1 

2 

-2 

-1 

-2 

1 

1 

2 

2 

-1 

-1 

-2 

Original  0| 
design 

Figure  7. 


0|  design 

(with  row  of  Os 
removed) 


Random 
permutation  of 

columns  of  0| 

design  (with  row 
of  Os  removed) 


Stacking  of  0|to  form  a  2:02  design. 


1 

-2 

2 

1 

0 

0 

-1 

2 

-2 

-1 

-2 

1 

1 

2 

2 

-1 

-1 

-2 

2:02  design 


“Complete”  designs  refer  to  designs  where  the  construction  includes  all  possible  p-way 
combinations  of  the  permutation  matrices. 


36 


The  orthogonal  and  space-filling  properties  of  the  best  of  five  s:0" 


designs  are  captured  in  the  Table  20. 


0" 

Number  of  Stacks  (s) 

n 

k 

Properties 

Original 
Matrix  (1) 

2 

5 

10 

15 

Mm 

1.369 

0.500 

0.354 

0.000 

0.000 

9 

4 

ML2 

0.054 

0.053 

0.054 

0.057 

0.058 

CL2 

0.184 

0.179 

0.173 

0.173 

0.174 

Mm 

1.785 

1.212 

0.829 

0.530 

0.559 

17 

8 

ML2 

0.314 

0.203 

0.156 

0.129 

0.128 

CL2 

0.314 

0.245 

0.196 

0.171 

0.167 

Mm 

2.417 

1.620 

1.395 

1.038 

1.104 

33 

16 

ML2 

11.525 

6.324 

3.211 

2.306 

1.828 

CL2 

0.834 

0.605 

0.403 

0.313 

0.262 

Mm 

3.342 

2.036 

1.952 

1.680 

1.809 

65 

32 

ML2 

1.72E+04 

1.03E+04 

5.22E+03 

3.51E+03 

2.98E+03 

CL2 

4.261 

3.043 

1.943 

1.385 

1.135 

Mm 

4.673 

3.225 

3.055 

2.660 

2.772 

129 

64 

ML2 

3.01E+11 

1.06E+11 

4.87E+10 

2.96E+10 

2.28E+10 

CL2 

98.993 

70.275 

44.563 

31.554 

25.775 

Table  21 .  Properties  of  s:0"  for  different  numbers  of  stacks. 


The  points  of  interest  on  the  effect  of  stacking  to  improve  the  space-filling 
properties  of  LH  designs  include: 

a.  Random  permutation  of  columns  being  stacked  provides 
improvements  to  the  space-filling  properties.  But  it  does  not  provide  the 
design  with  the  best  properties.  Total  enumeration  is  extremely  tedious  as 
the  total  possible  permutations  is  given  mathematically  as 

Total  permutations  =  (/f!)''s .  (4.1) 

b.  For  the  smaller  designs  (such  as  0|  and  O4 ),  the  Mm  property 

becomes  0  due  to  the  randomized  permutation  repeating  the  design  points 
due  to  the  stacking  of  additional  matrices.  The  other  space-filling 
properties  of  ML2  and  CL2  show  worse  performance  with  an  additional 
number  of  stacks,  possibly  due  to  the  same  reasons. 
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c.  For  the  larger  designs,  although  the  Mm  measure  becomes  worse 
for  all  cases,  this  is  due  to  additional  design  points  being  added  to  the 
design  matrix.  The  other  space-filling  properties  of  ML2  and  CL2  show 
improvement  in  performance  with  the  increased  number  of  stacks. 
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V.  CONCLUSIONS  AND  FUTURE  RESEARCH 


A.  CONCLUSION  OF  RESULTS 

From  the  OLH  designs  proposed  by  Ye  to  the  NOLH  designs  by  Cioppa, 
the  need  for  bigger  and  better  designs  can  be  seen.  Bigger  designs  allow  for  the 
design  of  larger  computer  simulations  and  experimentation.  This,  in  turn, 
enables  more  comprehensive  analysis.  The  inclusion  of  additional  combinations 
during  construction,  and  the  use  of  JAVA  for  construction,  extends  the  current 
catalogues  of  OLH  and  NOLH  designs  to  include  Og^^  and  designs. 

The  presence  of  discrete  variables  is  common  in  most  computer 
simulation  and  experiments.  This  thesis  provides  a  useful  guide  for  knowing 
which  columns  to  change  in  order  to  maintain  the  best  orthogonal  and  space¬ 
filling  properties;  how  the  effect  of  different  levels  affects  the  properties  of  the  LH 
design;  and,  how  the  use  of  positive-negative  coding  is  best  suited  for  2-level 
nominal  variables. 

The  space-filling  properties  of  LH  designs  are  improved  by  stacking  the 
designs.  The  LH  generator  in  JAVA  allows  for  the  creation  of  stacked  designs  as 
desired. 

B.  FUTURE  RESEARCH 

A  subsequent  literature  review  led  to  a  proposed  methodology  (Steinberg 
&  Lin,  2006)  for  creating  an  additional  increase  in  the  number  of  factors  for  an 
OLH  design.  However,  the  construction  method  could  not  be  found  to  access 
other  matrices  with  similar  constructs  to  compare  the  space-filling  and  orthogonal 
properties.  An  investigation  into  this  method  of  construction  for  the  LH  designs 
would  be  an  interesting  study,  to  determine  the  space-filling  properties  of  the 
designs  in  comparison  with  designs  by  Ye  and  Cioppa. 

Further  analysis  of  the  predictive  powers  of  the  extended  LH  design 
(which  includes  both  OLH  and  NOLH  designs)  is  suggested.  NOLH  designs 
have  orthogonal  properties  thresholds  for  selecting  candidate  designs.  It  would 
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be  interesting  to  discover  how  these  limits  affect  the  predictive  powers  of  the 
designs. 

Within  the  milieu  of  the  JAVA  application  to  generate  and  access  the 
properties  of  the  LH  designs,  a  smarter  and  more  efficient  method  to  search  the 
space  could  be  investigated  to  improve  the  speed  of  generating  the  design. 
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APPENDIX  A.  EXTENDING  THE  ORTHOGONAL  LATIN 
HYPERCUBE  FOR  M  =  4 


1.  Construction  of  M  for  the  OLH  Og^ 

The  e  vector  used  is  e  =  [  1  2  3  4  5  6  7  8]^. 

For  L  =  1,  A1e  =  [2  1  436587  where  A1  =  /0/0F?  yields 
the  following  permutation  matrix. 


0 

1 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

1 

0 

For  L  =  2,  A2e  =  [  4  3  2  1  8  7  6  5  where  A2  = /0R0F?. 

ForL  =  3,  A3e  =  [8  7  6  5  4  3  2  1  ]T  where  A3  =  R0R0R. 

Additional  permutation  matrices  are  constructed,  for  example  A1A3 
that  would  obtain  A1A3e  =  [7  85634  1  2  ]T,  yields  the  following 
permutation  matrix. 


0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

A1A2,  gives  A1A2e  =  [341  27856]^ 
A2A3,  gives  A2A3e  =  [56781  234]^ 
A1A2A3,  gives  A1A2A3e  =  [658721  43]^ 
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Thus  M  would  be  the  following  matrix. 


e 

Ai 

A2 

A3 

A1A2 

A1A3 

A2A3 

A1A2A3 

1 

2 

4 

8 

3 

7 

5 

6 

2 

1 

3 

7 

4 

8 

6 

5 

3 

4 

2 

6 

1 

5 

7 

8 

4 

3 

1 

5 

2 

6 

8 

7 

5 

6 

8 

4 

7 

3 

1 

2 

6 

5 

7 

3 

8 

4 

2 

1 

7 

8 

6 

2 

5 

1 

3 

4 

8 

7 

5 

1 

6 

2 

4 

3 

2.  Construction  of  S  for  the  OLH  Qi'^ 


From  the  original  matrix  with  So,  Si,  ...  ,  S3,  additional  columns  are 
generated.  These  additional  columns  are  just  multiplications  of  the  two  columns 
together,  and  include  S1S2,  S1S3,  S2S3  and  S1S2S3. 


So 

Si 

S2 

S3 

S1S2 

S1S3 

S2S3 

S1S2S3 

+  1 

-1 

-1 

-1 

+  1 

+  1 

+  1 

-1 

+  1 

+  1 

-1 

-1 

-1 

-1 

+  1 

+  1 

+  1 

-1 

+  1 

-1 

-1 

+  1 

-1 

+  1 

+  1 

+  1 

+  1 

-1 

+  1 

-1 

-1 

-1 

+  1 

-1 

-1 

+  1 

+  1 

-1 

-1 

+  1 

+  1 

+  1 

-1 

+  1 

-1 

+  1 

-1 

-1 

+  1 

-1 

+  1 

+  1 

-1 

-1 

+  1 

-1 

+  1 

+  1 

+  1 

+  1 

+  1 

+  1 

+  1 

+  1 

3.  Construction  of  T  for  the  OLH  Og^ 


Hadamard  product  of  M  and  S,  adding  a  row  of  Os,  and  the  mirror  image, 
results  in  the  design. 


Run 

Var  1 

Var  2 

Var  3 

Var  4 

Var  5 

Var  6 

Var  7 

Var  8 

1 

1 

-2 

-4 

-8 

3 

7 

5 

-6 

2 

2 

1 

-3 

-7 

-4 

-8 

6 

5 

3 

3 

-4 

2 

-6 

-1 

5 

-7 

8 

4 

4 

3 

1 

-5 

2 

-6 

-8 

-7 

5 

5 

-6 

-8 

4 

7 

-3 

-1 

2 

6 

6 

5 

-7 

3 

-8 

4 

-2 

-1 

7 

7 

-8 

6 

2 

-5 

-1 

3 

-4 

8 

8 

7 

5 

1 

6 

2 

4 

3 

9 

0 

0 

0 

0 

0 

0 

0 

0 

10 

-1 

2 

4 

8 

-3 

-7 

-5 

6 

11 

-2 

-1 

3 

7 

4 

8 

-6 

-5 

12 

-3 

4 

-2 

6 

1 

-5 

7 

-8 

13 

-4 

-3 

-1 

5 

-2 

6 

8 

7 

14 

-5 

6 

8 

-4 

-7 

3 

1 

-2 

15 

-6 

-5 

7 

-3 

8 

-4 

2 

1 

16 

-7 

8 

-6 

-2 

5 

1 

-3 

4 

17 

-8 

-7 

-5 

-1 

-6 

-2 

-4 

-3 
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APPENDIX  B.  CATALOGUE  OF  ORTHOGONAL  LATIN 
HYPERCUBE  DESIGNS 


The  designs  that  are  included  are  as  follows. 


Description 

Location 

Design  Matrix  forO^ 

Page  44 

Design  Matrix  for 

Page  44 

Design  Matrix  for  Ofg 

Page  45 

Design  Matrix  for 

Page  46 

Other  Design  Matrix  for  0“  to 

(^1025 

'^512 

SEED  Center  Website 
httD://harvest.nDs.edu/ 
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APPENDIX  C.  PROPERTIES  OF  LATIN  HYPERCUBES  WITH  DIFFERENT  CONSTRUCTION 


OLH 

NOLH  (SEED) 

1 

Mean  of  100  RLHS 

M 

fean  of  100 1 

RLHS-F 

k 

n 

cond{X^X) 

Pamp 

Mm 

ML2 

CL2 

cond{X^X) 

Pamp 

Mm 

ML2 

CL2 

cond{X^X) 

Pamp 

Mm 

ML2 

CL2 

cond{X^X) 

Pamp 

Mm 

ML2 

CL2 

7 

17 

1.00 

0.00 

1.48 

0.17 

0.26 

1.00 

0.00 

1.48 

0.15 

0.24 

9.53 

0.52 

0.97 

0.57 

0.45 

2.24 

0.23 

1.11 

0.19 

0.27 

8 

17 

1.00 

0.00 

1.79 

0.31 

0.31 

- 

- 

- 

- 

- 

14.41 

0.57 

1.12 

0.92 

0.53 

2.47 

0.24 

1.27 

0.35 

0.33 

11 

33 

1.00 

0.00 

1.67 

0.95 

0.41 

1.12 

0.02 

1.76 

0.73 

0.36 

8.44 

0.44 

1.28 

1.94 

0.56 

1.76 

0.14 

1.41 

0.83 

0.38 

15 

33 

1.00 

0.00 

2.20 

7.01 

0.73 

- 

- 

- 

- 

- 

16.64 

0.48 

1.72 

11.31 

0.96 

2.11 

0.17 

1.96 

6.23 

0.71 

16 

33 

1.00 

0.00 

2.42 

11.52 

0.83 

- 

- 

- 

- 

- 

19.73 

0.49 

1.83 

17.06 

1.08 

2.24 

0.17 

2.06 

9.94 

0.82 

16 

65 

1.00 

0.00 

1.79 

7.98 

0.72 

1.10 

0.02 

2.04 

4.46 

0.54 

6.47 

0.35 

1.60 

8.61 

0.73 

1.45 

0.08 

1.73 

4.80 

0.57 

22 

129 

1.00 

0.00 

1.79 

96.58 

1.39 

1.04 

0.01 

2.27 

37.78 

0.86 

4.65 

0.26 

1.97 

53.45 

1.03 

1.27 

0.05 

2.06 

37.26 

0.89 

26 

65 

1.00 

0.00 

2.70 

867.17 

2.25 

- 

- 

- 

- 

- 

15.50 

0.39 

2.53 

562.75 

2.42 

1.77 

0.10 

2.77 

431.13 

2.08 

29 

257 

1.00 

0.00 

1.81 

2.E+04 

2.92 

1.02 

0.00 

2.73 

482.91 

1.44 

3.61 

0.20 

2.35 

488.63 

1.61 

1.17 

0.03 

2.42 

392.36 

1.48 

31 

65 

1.00 

0.00 

3.19 

1.E+04 

3.816 

- 

- 

- 

- 

- 

22.84 

0.40 

2.96 

4.E+03 

4.33 

1.97 

0.12 

3.26 

4.E+03 

3.77 

32 

65 

1.00 

0.00 

3.34 

2.E+04 

4.261 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

37 

513 

1.00 

0.00 

1.86 

5.E+04 

6.81 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

42 

129 

1.00 

0.00 

3.27 

2.E+06 

11.34 

- 

- 

- 

- 

- 

11.26 

0.30 

3.44 

2.E+05 

10.18 

1.54 

0.07 

3.68 

2.E+05 

9.32 

46 

1025 

1.00 

0.00 

1.91 

2.E+06 

18.24 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

57 

129 

1.00 

0.00 

4.14 

5.E+09 

49.27 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

63 

129 

1.00 

0.00 

4.56 

1.E+11 

89.03 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

64 

129 

1.00 

0.00 

4.67 

2.E+11 

98.99 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

64 

257 

1.00 

0.00 

3.88 

2.E+11 

131.73 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 
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OLH 

NOLH  (SEED) 

1 

Mean  of  100  RLHS 

M 

fean  of  100 1 

RLHS-F 

k 

n 

cond{X'^X) 

Pamp 

Mm 

ML2 

CL2 

cond{X^X) 

Pamp 

Mm 

ML2 

CL2 

cond(X^X) 

Pamp 

Mm 

ML2 

CL2 

cond{X^X) 

Pamp 

Mm 

ML2 

CL2 

93 

513 

1.00 

0.00 

4.52 

1.E+18 

4.E+03 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

99 

257 

1.00 

0.00 

5.25 

3.E+19 

5.E+03 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

120 

257 

1.00 

0.00 

6.13 

2.E+24 

3.E+04 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

127 

257 

1.00 

0.00 

6.49 

9.E+25 

6.E+04 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

128 

257 

1.00 

0.00 

6.57 

1.E+26 

7.E+04 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

130 

1025 

1.00 

0.00 

5.12 

2.E+27 

5.E+05 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

163 

513 

1.00 

0.00 

6.52 

9.E+34 

9.E+06 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

219 

513 

1.00 

0.00 

8.05 

1.E+48 

2.E+09 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

247 

513 

1.00 

0.00 

8.91 

3.E+54 

2.E+10 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

255 

513 

1.00 

0.00 

9.21 

2.E+56 

5.E+10 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

256 

513 

1.00 

0.00 

9.26 

3.E+56 

5.E+10 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

256 

1025 

1.00 

0.00 

7.93 

7.E+57 

9.E+11 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

382 

1025 

1.00 

0.00 

10.35 

5.E+87 

4.E+17 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

466 

1025 

1.00 

0.00 

11.99 

2.E+107 

9.E+20 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

502 

1025 

1.00 

0.00 

12.79 

3.E+115 

2.E+22 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

511 

1025 

1.00 

0.00 

13.04 

3.E+117 

4.E+22 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

512 

1025 

1.00 

0.00 

13.08 

5.E+117 

4.E+22 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 
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